Human embryo evaluation using ai/ml analysis of real-time video for predicting female-sex offspring

ABSTRACT

A computer-implemented method for predicting the likelihood an embryo will produce a human female offspring by processing video image data derived from video of a target embryo. The method includes receiving image data derived from video of a target embryo taken at substantially real-time frame speed during an embryo observation period of time. The video contains recorded morphokinetic movement of the target embryo occurring during the embryo observation period of time. The movement is represented in the received image data and the received image data is processed using a model generated utilizing machine learning and correlated embryo outcome data to predict the likelihood the target embryo will produce a human female offspring.

CROSS REFERENCE TO RELATED APPLICATIONS

The present patent application is a continuation of PCT/US2021/044423filed Aug. 3, 2021, designating the United States and which claims thebenefit of U.S. Provisional Patent Application No. 63/060,3554, filedAug. 3, 2020. Each of the aforementioned patent applications isexpressly incorporated herein by reference, in its entirety, withoutdisclaimer.

SUMMARY

Combining state-of-the-art artificial intelligence (AI), machinelearning (ML), video image data processing and optics systems, practicalinsights from clinical embryologists and scientific input from dataengineers, the present technology provides the world's mostcomprehensive embryo evaluation system for detecting and/or predictingembryo morphokinetics, viability and developmental potential in asingle, easy to use system. The technology processes digital video imagedata from real-time video taken of an embryo proximate in time beforetransfer or storage such as by cryopreservation. The image data of theresulting signal is then processed to reveal information, heretoforehidden, to the operator and other interested parties. With thistechnology, model-processed real-time video image data of an embryo canpredict, among other things: (1) the embryo's present viability at timeof transfer, (2) the embryo's inviability at time of planned transfer,(3) the embryo's likelihood (qualitatively, quantitative and/orprobabilistic) to result in pregnancy and produce live offspring, (4)the embryo's likelihood of producing a male offspring, (5) the embryo'slikelihood of producing a female offspring, (6) genetic inferiority(likelihood to embody disease or genetic issue, and of what type) of theembryo and (7) genetic superiority (likelihood to produce desired traitsin offspring) of the embryo. The instant deep learning model(s) can alsorank a number (group) of embryos based on one or more predictive embryocharacteristics or predicted offspring traits based on generatedrelative strengths (probability of occurrence) of the predictedcharacteristic among the group of blastocysts enabling an embryologist'sselection and transfer of the most desirable embryos into recipient(s).

In one embodiment, video image data is utilized to develop deep learningmodels that provide the basis for an AWL-based software package thatenables a non-invasive, quantitative, non-subjective system and methodto identify high-quality, viable embryos likely to result in pregnancyand healthy offspring. Currently, embryologists have limited ability toevaluate embryo health and viability, making it all too common tounknowingly transfer dead or otherwise inviable embryos into recipients.The present technology allows a user to make more informed decisionsabout which embryo(s) to transfer and reduces the transfer oflow-quality embryos which will not establish pregnancy. Technologiessuch as this that improve pregnancy outcomes of IVF beneficially affectpatients both physically and psychologically, as well as having apositive financial impact on the industry as a whole through costreduction. Another aim of this technology is to enhance the workflow ofthe user by enabling a single multi-embryo video/scan to provideimmediate feedback through real time image data collection andprocessing. In the case of animal breeding and production, thetechnology is utilized to select the highest quality embryos that aremost likely to achieve pregnancy and produce disease-free offspring thatembody the most superior of desired traits, including sex.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a graph showing that morphokinetic activity is predictive of apreimplantation embryo's likelihood to produce pregnancy;

FIG. 2 is a histogram depicting morphokinetic patterns of embryos thatproduced pregnancy;

FIG. 3 is a histogram depicting the range of embryo movement withinsubzonal space as being an indicator of embryo competency for producingpregnancy;

FIG. 4 is a graph showing that high morphokinetic activity is predictiveof a preimplantation embryo's likelihood not to produce pregnancy;

FIG. 5 illustrates components, area measurements and axial lines ofmeasurement on a blastocyst that can be utilized to describe theblastocyst's morphokinetic activity;

FIG. 6 superimposes lines of measure on an embryo for demonstratingmovement of the inner cell mass within the zona pellucida of ablastocyst (an embryo) by assessing the varying distances therebetween;

FIG. 7 depicts a series of 12 frames extracted from a video clip of ablastocyst using a microscope at 150× magnification;

FIG. 8 depicts the same 12 frames of FIG. 7, but after being filteredusing motion magnification software;

FIG. 9 depicts the outcome of video data processed using a model trainedin accordance with the presently disclosed technology and whichdemonstrates: (1) a prediction accuracy rate of 91% for blastocysts thatwill not achieve pregnancy upon transfer and (2) a prediction accuracyrate of 100% for blastocysts that will achieve pregnancy upon transfer;

FIG. 10 is a schematic of an exemplary deep learning neural networkuseable for presently disclosed AI/ML model training, model generationand video image data processing using the generated models forpredicting embryo traits and produced offspring characteristics;

FIG. 11 depicts an exemplary processor-based computing system suitablefor use as a platform for executing certain routines of the presentlydisclosed AI/ML modeling; and

FIG. 12 represents a goal of the presently disclosed technology which ispredicting an embryo's competence to establish pregnancy and produce thelive birth of a desired offspring as described in the presentdisclosure.

DETAILED DESCRIPTION

Users of the presently disclosed technology include reproductiveendocrinologists, specialized OB/GYN's, embryologists, veterinarians,animal breeders and trained laboratory personnel, among others. In therealm of human reproduction, users of this technology include embryologylab directors and clinical embryologists who are practicing AssistedReproductive Techniques (ART). ART patients, and in particular IVF (InVitro Fertilization) patients, are prime influencers of this technologyas they play a key role in selecting which fertility treatment(s) toemploy that best assures a live birth out of their scarce and fragileinventory of gametes/oocytes/embryos. As a precursor to thistechnology's development, customer interviews were conducted with 231interviewees during the 2018 National Science Foundation I-CorpsNational Atlanta Winter Cohort regarding needs in the industry. Based onthe results of those interviews, the perspective became clear that therehave not been any transformative technologies to predict embryoviability in nearly four decades.

In practice, clinical embryologists and/or otherwise trained personnelare responsible for selecting the embryo to be transferred into therecipient and that responsible person often unknowingly selects andtransfers non-viable embryo(s) into the recipient due to lack ofaccurate means to evaluate embryo health and quality. Embryologists' andfertility clinics' reputations are predominantly based on publishedpregnancy success rates, so there is active competition for the highestrankings, as high rankings best produce revenue. Poised for success,this technology is in a class of its own as the only embryo assessmenttool that is NOT subjective (non-subjective), NOT invasive(non-invasive) and enables real-time assessment of presently existingembryo morphokinetics in both human and non-human mammals.

Infertility in humans is classified as a disease by the World HealthOrganization and affects 48.5 million couples worldwide. In VitroFertilization is presently the most effective treatment of infertility,but the produced livebirth rate in 2018 was only 26%. Unfortunately, IVFis renowned for being expensive, stressful and generally unsuccessful,and which more often than not, delivers a disappointing experience.

On the positive side however, ART is expanding the relevant market as itis no longer just a treatment of infertility but has the added thecapability to extend fertility to others in many different situationsthough cryopreservation of eggs, sperm and embryos. One example is thepreservation of healthy cells/gametes of cancer patients prior toundergoing chemotherapy in the event the patient is not able to producethem after such treatment. ART also enables single individuals andhomosexual couples to enjoy biological families, which might otherwisenot have been possible. For these reasons, as well as others, ART isbecoming a routine method to start or extend a family, but improvementsare definitely welcome.

Since the birth of Louise Brown in 1978 by IVF, over 10 million babieshave been born via IVF and Intracytoplasmic Sperm Injection (ICSI),worldwide. In the United States, babies born from IVF/ICSI comprisenearly 2% of the annual birth count. For IVF, a woman is givensuper-ovulatory hormones to encourage more eggs to develop in theovaries at once. The woman's follicular development is closely monitoredto time the egg retrieval in which the fertility specialist aspiratesthe eggs from the ovary. Each egg is fertilized with sperm and thefertilized eggs are permitted to develop into embryos over the course offive to six days. It is likely that multiple embryos will develop intoseemingly healthy blastocysts. From them, the embryologist isresponsible for choosing which embryo(s) to transfer into the patient.Any remaining embryos are typically frozen and can be used forsubsequent transfer attempts. This process is referred to as a “round”of IVF and generally costs about $23,000. If a child is not produced inthis first round, the patient can opt to undergo a frozen transfer, inwhich one or more of the remaining frozen embryos is thawed andtransferred into the patient. Frozen embryos can be transferred until nomore are available, in which case the entire process must start over. Onaverage, patients undergo 2.7 rounds of IVF and spend $60,000 to achievea pregnancy that survives to term. Unfortunately, a major contributor tothe low success rate of IVF is an inability to discern and choosehealthy, viable embryos at the time of transfer.

There are approximately 450 IVF clinics in the United States, as well asdonor egg banks that collaborate with over 150 of the IVF clinics. Outof those clinics, 271,398 human embryo transfer procedures wereperformed in 2018. Approximately 50% of those IVF procedures in womenages 35 and under resulted in a live birth. For women ages 42 and older,the number plumets to 3.9% of egg transfers that result in a birth. Withthat as background, it is clear that any technology that enablesembryologists to select embryos most likely to produce a healthy birthand/or survive and thrive after cryopreservation has great prospect foradoption by the industry. Just considering human embryo transfers in theUnited States (excluding animal transfers), it is calculated that a 3%improvement in pregnancy outcomes would currently provide over$187,000,000 in annual patient savings. Additionally, technologies likethis that improve pregnancy success on a per-transferred-embryo basislessens the need to transfer multiple embryos into a single recipientwhich reduces the risk of undesired, multiple births. This clearlyreduces costs, but equally important, it reduces complications tomothers and babies from simultaneous gestations.

Embryo screening is limited by the requirement that embryos must survivethe diagnostic process, unharmed. Many existing assessment methods suchas staining and electron microscopy are only used in a research settingbecause the processes are lethal. The most commonly used method toassess the quality of embryos and oocytes for transfer is amorphological analysis using a light microscope. In this assessment, thetechnician examines the cells for visually apparent characteristics suchas cell shape, size, symmetry and color. While this method isnoninvasive and cost effective, it is subjective, relying solely on theexaminer's discretion and does not include information about an embryo'sbiochemical content or genetic make-up. As an aid to morphologicalanalysis, time lapse imaging systems, such as the EmbryoScope®, areavailable that link developmental events with the elapsed time at whichthe different events occur. However, in a recent report published in theJournal of Reproduction and Genetics, only 17% of surveyed labs had atime lapse imagining system. Moreover, a majority of surveyed labdirectors responded that “time lapse imaging will not become thestandard of care because the technology is too expensive, there is noevidence it provides additional benefit and it is too time intensive andimpractical.”

Today, genetic assessment requires a biopsy of cells from the embryo.The biopsied cells are sent to an off-site lab for genetic testing andthe left-behind embryo must be frozen until lab results are received.Pre-implantation Genetic Testing (PGT) encompasses both pre-implantationgenetic screening and pre-implantation genetic diagnosis, which caninform embryologists and patients of genetic abnormalities (such asaneuploidy or trisomy), genetic disease and embryo sex. PGT was designedfor high-risk patients with a known condition or genetic trait, and notfor the average IVF/ART patient. Disadvantages of PGT are its invasivenature, increased risk of cell death and it requires expensive equipmentand highly trained personnel. The costs range from $4000-$7500, whichare in addition to the base fees associated with IVF procedures. PGTdata suggests there is no difference in pregnancy rate for patientsunder the age of 36 years old. Other reports claim PGT significantlyincreases pregnancy outcomes, but many physicians warn that it alsoincreases the incidence of miscarriage, which indicates that the “takehome baby rate” is likely unimproved. In contrast, the presentlydisclosed non-invasive embryo analysis system provides a newly-availablesuperior standard of care, easily incorporated into current laboratoryprotocols and procedures.

The solutions made possible by the present technology will quicklypenetrate the $6.2 billion US market of the human clinical infertilitysector. It also easily scales to service the $37.7 billion globalinfertility market. With fewer than half of the people suffering frominfertility seeking care, technologies such as the present that reducecost and increase success rates will undoubtedly expand and grow thealready substantial infertility care market. As an example, onepublished market analysis indicates that a 50% reduction in the price ofIVF services would translate into a 160% increase in the utilization offertility services.

Turning now from the human experience to the animal sciences, it isnoted that the global population is expected to reach 9.5 billion by2050, an approximately 20% increase over 2020. It is projected that thedemand for animal derived protein will at least double, resulting inconcerns over food security and sustainability. Current agriculturalpractices are already placing tremendous pressure on the earth's finiteresources and are largely responsible for a vast proportion ofgreenhouse gas (GHG) emissions. Well managed cattle (also referred toherein as cows and/or bovine) production helps to meet food security andenvironmental goals, as beef and dairy products both providenutrient-rich, high-quality protein to consumers. Cattle areparticularly suited to meat production as they are a robust animal thatadapt well to changes in climate and can graze on pastureland generallynot suitable for crop production due to climate, soil and topographiclimitations. Cows convert forage into high quality protein and excretefertilizer as a beneficial by-product. Modern cattle productionpractices have been adapted to reduce the industry's carbon footprintand as a result methane output has declined approximately 40% since1980. Genetic selection through progressive breeding strategies canfurther reduce GHG emissions from cattle production by improving animalefficiency, improving the feed conversion ratio, reducing dry matterintake and reducing enteric methane production. Selection for geneticsuperiority in cattle, among other animals, is enabled by the presentlydisclosed technology in which an appropriately trained AI/ML model(s)predicts the likelihood that a particular embryo will produce offspringhaving desired characteristic(s) based on processing digital video imagedata extracted from a real-time, short duration video typically takenright before transfer or storage (cryopreservation) of the embryo.

As stated above, methane (CH4) is the major green-house gas emitted byruminant production systems with CH4 from enteric fermentationaccounting for 12% to 17% of that emission. Diet is a major factor(roughage and concentrate) contributing to methane emissions fromruminants. Dietary aspects known to reduce ruminant methane productioninclude decreasing the proportion of cereal grains and legumes in theanimals' diets and adding plant species that contain secondarymetabolites such as tannins and saponins which affect methanogenesis inthe rumen and reduce ruminant methane emissions. Effectiveness offeedlot programs implementing dietary improvements has improved thecarbon footprint of beef production in Canada to as low as 17 kg ofcarbon dioxide equivalents (CO2e) per kg of food for feedlot finishedbeef as compared to grass-finished beef in Brazil that emit as much as40 kg CO2e per kg of food. Despite the known advantages of certainplants, grassland renovation is often not practical as many of theseplant species are weak competitors compared to native grasses andconsequently is all-too-often not affordable or sustainable. Therefore,the most practical and rapid mitigation procedure is to reduce the percow CH4 emission through animal breeding and genetic selection for feedefficiency, as those effects are permanent and cumulative.

Genetic selection utilizing the present technology can reduce GHGemissions from beef and dairy cattle production systems because it: (1)enables increased production per animal which reduces the number ofindividual animals to produce the same amount of beef and dairy; (2)lowers emissions per unit of beef or dairy produced; and (3) reducesconsumption of grain, concentrates, roughage, water and land. Embryotransfer and in vitro fertilization can further perpetuate genetictraits of superior animals because they enable genetically superioranimals to produce more offspring in a single year than can be achievedin nature. Additionally, ART can help producers control the sex ratio tobe favorable for the most desired sex for the operation. As an example,dairies have a preference for female cows because they produce milk andmales do not. Potentiated bovine embryo transfer provides other economicadvantages such as decreased calving intervals, more consistent calfcrop production and maximized product value. Due to these benefits, ETand IVF have become routine breeding strategies in livestock operationsand approximately 2.5 million cattle embryos are transferred in NorthAmerica each year.

To perform ET or IVF currently, superior female animals referred to as“donors” are stimulated with superovulatory hormones to produce anincreased number of oocytes. The oocytes can be aspirated from the ovaryby a veterinarian and fertilized in vitro or the animal can beartificially inseminated and fertilized embryos can be flushed from theuterus six to seven days later. Today, the veterinarian or embryologistexamines the embryos under a light microscope and assigns a grade basedon visible morphological characteristics. Highly graded, grade-1 andgrade-2 embryos are transferred into recipients or frozen for use at alater date.

Unfortunately, the success rates of ET and IVF in cattle are still verylow. The success rate (pregnancy achieved) of ET today is less than 65%in cattle, and the success rate of conventional IVF is less than 30% incattle. While causes of failed pregnancy are multi-factorial and canstem from embryonic, maternal and/or environmental stressors, it isestimated that 20% of transferred embryos are actually non-viable at thetime of transfer and will never result in pregnancy.

Technology that could increase pregnancy rates and optionally controlsex ratio would be extremely valuable. For cattle, approximately $1200is invested into each embryo transferred, regardless of whether itestablishes a pregnancy or produces a calf of desired sex. With currentpregnancy outcomes at less than 65% for conventional embryo transfer andless than 30% for IVF, technology such as that presently disclosed thatimproves pregnancy outcomes and provides significant savings toproducers will be quickly adopted. Additionally, enabled selection forgenetic superiority and desired sex adds further value. ET coupled withthe use of the present technology is the fastest way to change andimprove the genetics in a herd and maximize beef producer and dairyprofitability.

The presently disclosed technology facilitates the provision of acomprehensive, portable platform for obtaining, on-site, real-timeembryo video clip(s), and while still on-site, processing the obtainedvideo image data using an appropriately trained AI/ML model to makerelevant predictions about a target embryo(s) and/or offspring producedtherefrom.

Aspects of the present technology that provide a competitive advantageinclude: (1) its non-invasive nature, (2) it is not subjective, (3) itutilizes quantitative analysis that incorporates artificial intelligenceand deep learning algorithms, (4) it evaluates and/or predicts embryogrowth, health, sex and many other embryo characteristics and/or traitsof offspring produced therefrom by processing image data from real-timevideo and provides immediate diagnostic feedback (in minutes, if notseconds) to the user at an affordable price. This technology subjectsthe target embryo to essentially no additional risk, does not requirebiopsy or manipulation of cells and facilities the evaluation ofmultiple embryos at one time resulting in predictive data beingavailable for each embryo in minutes, if not less. In an alternateconfiguration, the presently disclosed technology enables the user to“see” changes in embryo morphokinetics within the same time frame, ifthe diagnostic image data is optionally processed formagnification/amplification as herein described.

At different times, real-time video image data has been collected fromanywhere between 150 and 1000 bovine embryos for any one study.Real-time digital video image data is collected and processed from thebovine embryos, together with correlated pregnancy and/or offspringcharacteristic/trait data (correlated outcome data). The video fromwhich the real-time video image data is derived must contain a videoresolution sufficient to record morphokinetic movement of the targetembryo. From the video image data, changes in embryo morphokinetics areobserved in substantially real time. Another characteristic of thetarget embryo evidenced in the received image data is elasticity of theembryo's outer wall. Utilizing the correlated outcome data for modeltraining, predictive model(s) have been generated regarding producedpregnancy and offspring outcomes.

According to this disclosure, the origin of the digital video image dataused for both AI/ML model training and predictive processing isreal-time video taken of pre-implantation embryos that are typically inthe blastocyst or morula stage of development, and which occurs betweenabout five to nine days, post fertilization. The embryos can be theproduct of either in vitro (outside the donor) fertilization (“IVF”) orin vivo (within the donor) fertilization by artificial insemination. Inthe case of in vivo fertilization, the produced embryos are subsequentlyflushed from the host, typically for transfer into multiple individualrecipients. In this specification, references to “embryo” include atleast blastocyst and morula stage embryos and, vice versa, regardless ofthe embryo being the product of in vitro or in vivo fertilization.

For taking the referenced video(s), one or more embryos are placed in adisposable, typically polystyrene container (dish) compatible with theemployed video image and diagnostic equipment. Advantageously, suchreceiving dish can have a unique QR code compatible with the analyticsoftware for embryo identification. In one configuration, the dish cancomprise (include, but not be limited to) a plurality of pre-labeledwells, each intended to receive one embryo, and which among otherthings: (1) aids proper placement of the embryos in the technician'sand/or camera's field of view, (2) provides an identify to each ofseveral contained embryos, and (3) correlates generated predictions tothe respective embryos.

The real-time digital video clip(s) can be taken with a variety of videoimaging equipment including, among others, digital cameras, actioncameras, smartphone cameras, tablet cameras, board cameras and the like.In the instance of board cameras, also called printed circuit board(PCB) cameras, the included optical devices and image sensors aremounted directly on a circuit board, as is often the case insmartphones. An associated display can be optionally provided to a boardcamera, as is the case with a smartphone, and the image signals arerelayed through an I/O of the PCB. As there are no analog controls,recording options are also controlled through the interface.

It is desirable that each real-time video clip comprising image dataused as training data (and subsequently, video-based target image datato be analyzed using resulting so-trained models) be relatively short.For example, the video clips advantageously extend from around fifteenseconds up to several minutes long in duration. It is not necessary thateach clip be individually recorded. Multiple short duration clips can beexcised out of a longer blastocyst video. The duration of the trainingclip is selected to correlate best to video image data that will beprocessed by the resulting model; therefore, utilizing training datasource clips of similar duration subsequently enables more accuratepredictions from the generated model.

Using appropriately trained model(s), assessment is made of ablastocyst's movement recorded in one of these video clips of relativelyshort duration. The duration of the video clip is preferably 30 secondsor less (5, 10, 15, 20, 25, 30 seconds), but the duration of the clipcan be as much as one minute, two minutes, 5 minutes or more, thoughsuch longer clips are less desirable given their increased amount ofconstituent data. Video described in this disclosure is referred to asreal-time video, as compared to time-lapse video that is historicallyknown for use in embryo evaluation and assessment. Time-lapse “video” ismade-up of a series of consecutive still photos or “frames” that havesubstantial time-spacing (30 seconds or more, but typically minutes orhours) therebetween. Heretofore, relatively short duration, real-timevideo recordings of embryos have not been analyzed regarding existingembryo short, rapid morphokinetic movement because none was visiblydetectable, even under typically employed microscope magnification.

In fact, as described herein, significant morphokinetic movement hasbeen revealed to be occurring in observed blastocysts over the shortvideo time periods. In accordance with the present disclosure, thismovement is recorded using real-time video taken via a microscope at amagnification power (e.g., 150×) heretofore accepted as only beingsuitable for observing a “static” embryo (i.e., a “snap-shot view”).This was a reasonable deduction because technician-observation of ablastocyst under microscope magnification for the same amount of time asvideo is now taken does not reveal the morphokinetic movement that isoccurring because even under microscope magnification, that movement isstill humanly imperceptible. Now knowing that the blastocyst'smorphokinetic movement is recorded in the image data of these real-timevideos as described herein, even though not perceptible, the image datais suitable for model training and/or predictive processing usinggenerated model(s).

The presently described real-time video clips each comprise a series ofconsecutive, equal time-spaced digital image frames, and the duration ofthe time spacing is referred to as frame speed. A recorded blastocystmorphokinetic movement traverses a short distance, at a fast rate, andtherefore has a commensurately short duration. Consequently, the framespeed utilized for the recordings should be of similar duration, andpreferably faster. Exemplarily, frame speeds of ten frames-per-second(10 fps) and fifteen frames-per-second (15 fps) have been utilized tocapture blastocyst movement. In practice, faster frame speeds haveproven beneficial, particularly in training stages of the presentlydisclosed AI/ML analysis because the quick, short-distance morphokineticmovements made by the blastocyst are captured with greater accuracy,clarity and fidelity when faster frame speeds are used. Conversely, asvideo frame speed decreases and there is a greater time gap betweenconsecutive frames, the blastocyst movement becomes less perceptible. Atspacings greater than about two frames per second (2 fps), individualmorphokinetic movements of the blastocyst can become unobservable. Whenthe duration of a morphokinetic movement is less then the frame speed,that movement can be “lost” in the gap-space between two frames that“straddle” that movement, especially if the movement is a reboundingmotion that at first occurs in one direction, and then elastically“snaps back,” all within the gap-space between two straddling frames.

Whether being prepared as training data or as target data to be analyzedusing generated model(s), preprocessing of the blastocyst image data canadvantageously include normalizing the “raw” image data to that of a“standard,” comparable blastocyst. Though various known methods can beused to accomplish such normalization of video image data, here, thedesired end result is to make each “image” of each blastocyst comparableto one another, with minimized “noise” caused by different setups usedto take the various original videos. One example is to adjust each“image” of each blastocyst to the same “size” by scaling up (expanding)or scaling down (shrinking) the respective images to a “standard size.”

In some instances, the view area of the videos is normalized to be thesame, but not the size of each blastocyst image. This type ofnormalization is used when relative size drives a predictable outcome.For instance, relative size can be predictive of blastocyst viabilityand sex. Relatedly, relative blastocyst size can also be a determinativeor desirable differentiating factor amongst a group of co-fertilizedblastocysts (multiple eggs fertilized at the same time and grown for thesame time period) that are being model-assessed for viability and/orsex, and relative size is predictive, or can at least be used to rankthe blastocysts for certain predictable pregnancy characteristics and/oroffspring traits amongst the group of embryos.

During the training process, a model can be trained to predict amongother things about the embryo: viability, likelihood to producepregnancy, likelihood to result in a live birth, offspring sexprobability (how likely to produce a male offspring or how likely toproduce a female offspring), and the presence or absence of geneticanomalies and disease, such as diabetes, cancer, other cell growth andthe like or the capability for passing along desired characteristic(s)to offspring, such as highly efficient feed utilization.

As an adjunct, embryo video image data processing has been employed inaccordance with the present disclosure to observe and evaluate embryomovement responsive to an induced environmental stressor and determinedthat the observed motion is predictive of related outcomes (good/bad)that are embodied in resulting embryos and/or offspring. This can beuseful to determine early prognosis of cancer, metabolic profile,genetic defects and the like. Studying embryo homeostasis mechanisms byenabling visualization and/or other analysis of embryo response(typically movement) to environmental changes has been determined to beuseful to optimize culture systems, media, materials, nutrients, and toevaluate cellular metabolites, ambient environment impacts (light,temperature, pressure and the like) and overall developmentalcompetency. Relatedly, the development/testing of vaccines have beenaided by the present technology. It has been noted that embryosexperience predictive morphokinetic motion when inoculated with certainpathogens. It has also been observed that pathogen-carrying embryos(infected, but otherwise unaffected) experience predictive morphokineticmotion when exposed to vaccines, typically being tested for efficacyagainst the carried pathogen.

As an example of that described above, it is known that cystic fibrosis(CF) is caused by a genetic mutation in the CF Transmembrane ConductanceRegulator gene (CFTR gene) that regulates the movement of chloride inthe body. Prospective parents can be screened to determine their carrierstatus, but the only way to determine whether the genetic mutation hasbeen passed along to an embryo is through pre-implantation geneticdiagnosis via biopsy. The expression of CF is an inability to regulatesalt and water transport across epithelial tissue. In accordance withthe teachings of the present disclosure, processed embryo video imagedata revealed that embryos embodying the CF genetic mutation experienceddetectible reactive motion when exposed to chloride. In this case,clinical exposure was created by adding chloride-containing material(salt) to the liquid media containing the embryo. When an embryo carriedthe CFTR gene mutation, the addition of chloride stressed the embryo andcaused responsive observable motion. In accordance with instantteachings, an appropriately trained model was used to process digitalimage data from video taken of the potentially affected embryo andpredicted whether the CFTR gene mutation was present. The prediction wasmade in terms of the likelihood (probability) that a resulting offspringwould suffer cystic fibrosis. Conversely, the videos of embryos that“tolerated” the addition of chloride to their containing media showedmuch less reactionary motion. The video image data of that motion wasprocessed using the trained model and a likelihood was predicted thatresulting offspring would not embody the CF genetic mutation. Insummary, AI/ML models generated according to the teachings of thepresent disclosure have been demonstrated to competently process digitalimage data sourced from video taken of embryos responding to an inducedstimuli indicative of the presence of a gene mutation in an embryo andpredict the likelihood resulting offspring will embody the mutation.

The protocol and process for predicting the presence of most genetic andmetabolic diseases embodied in an embryo, and the likelihood of itsexpression in resulting offspring, follows that which is described abovefor cystic fibrosis. For example, reactionary motion by an embryo toglucose added to the containing media predicts diabetes, while potassiumpredicts Hyperkalemic Periodic Paralysis in horses, among others.

As intimated above, the present technology facilitates non-invasiveembryo sex selection. There is scientific evidence indicating that maleembryos develop faster than female embryos. This is observed by totalcell count at the blastocyst stage of development. The data ispredominantly anecdotal because it is difficult to count cellsnon-invasively without harming the embryo. However, for there to begreater male cell count over a predetermined period of time, the numberof cells in “male” embryos must multiply at a faster rate. In accordancewith the present teachings, processing video image data utilizingappropriately trained models for predicting respective sex can beutilized to determine the likelihood that the target embryo will producea male offspring or will produce a female offspring. In this regard,even though biological sex can be considered binary, because a modeltrained for predicting “female” offspring indicates a 70% likelihood ofproducing a female offspring does not necessarily mean that acorrespondingly trained model for predicting “male” offspring willindicate a 30% likelihood of producing a male offspring. To fit userexpectation, however, an exemplary resolution is to predict bothlikelihoods, but only express a preference when one sex is predictedwith a likelihood greater than 50% and the other sex is predicted with alikelihood less than 50%. Instances such as this help to illustrate thebenefits and uses of the present technology, but also highlight theimportance of human review and participation in embryo transferdecisions.

Typically, the videos of the blastocysts to be analyzed as describedherein are taken in the “field,” which is considered to be wherever theblastocyst is evaluated for transfer or preservation. In this context,“field” when referring to animals can be a veterinary clinic, barn,stock trailer and/or otherwise outdoors, for example. When consideringhumans, “field” typically refers to a fertility clinic and the like.Therefore, to avoid the need to transmit the video (which can comprise adata file of substantial size) to a remote analysis site, the trainedmodel can be located on a local processor (computer, specialized chip,tablet, smart phone and the like) where the desired predictions aremade. Alternatively, the data file can be transmitted to a host's remotesite where the video file is processed, and resulting predictionstransmitted back to the user in the field.

In practice, regardless of the field in which the blastocysts arecollected, it is common for a plurality of blastocysts to be collectedand suspended in nutritional media in a disposable receptacle such as aplastic or glass petri dish, organ culture dish or multi-well plate, asexamples. Accordingly, a single video of a plurality of blastocysts inthe receptacle can be taken and processed regarding each of theindividual blastocysts for the characteristics which the model has beentrained. An important aspect of the receptacle is that it has atransparent, substantially flat floor. An inverted microscope isexemplarily utilized to record a substantially distortion-free video ofthe blastocyst(s) from below the receptacle, through its clear, flatbottom surface. This creates a “goggle effect” and reduces “noise” inthe sample from wave refraction caused by a rippling upper surface ofthe liquid media if observed from above. Exemplarily, an Iolight® brandcompact inverted microscope has been utilized to record a video of theblastocyst(s). The magnification power is one-hundred and fifty times(150×), the frame speed is 10 fps, the pixel density is 2592×1944 andthe microscope can connect to smartphones and/or tablets for sharingproduced videos.

The presently disclosed technology enables the user to collect orreceive secondary data obtained close in time to when the video fromwhich the real-time digital video image data is derived was taken. Inone configuration, the presently disclosed technology enables the userto perform a specific gravity assessment of the target embryo within onehour, or less, of taking the video of the target embryo. The specificgravity assessment used to obtain this example of secondary data isdependent upon a descent rate of the target embryo through embryoculture media. Secondary data collected and processed at the proximatetime as real-time digital video image data can be indicative of thetarget embryo's viability, among other things.

In one exemplary implementation illustrated in FIG. 9, a convolutionalneural network model was utilized that is known to persons skilled inthe art as Xception, or Xception71, and which is seventy-one layers deepand can consider approximately twenty-three million parameters. In thisexample, the model was trained on 256 videos of bovine embryos. Eachvideo was thirty-five seconds in duration, taken at a frame-speed offifteen frames-per-second, thereby rendering 134,400 (256×35×15) framesper processed video. Supervised training of the model was conducted withthe 256 videos and each video had pregnancy data associated therewithindicative of whether a pregnancy (bred) occurred or not (open).Xception was used to process the training data and generate a predictivemodel. Thereafter, upon processing twenty new videos (n=20) using thetrained model, ten of eleven (90.9%) videoed embryos were accuratelypredicted to NOT establish a pregnancy (open). The model accuratelypredicted nine of nine (100%) videoed embryos would establish apregnancy (bred).

As an illustrative example, consider a dairy operation in which tenblastocysts have been flushed from a donor cow and deposited into apetri dish. There are only five available recipient cows, so thetechnician wishes to identify the five most viable, female-producingblastocysts. Video of the ten blastocysts is processed by a previouslytrained model that predicts and assigns, on an individual basis, alikelihood the particular blastocyst will produce a female calf. Thecomparable predictions (embryo outcome score) are then communicated backto the technician visually, or otherwise. In this way the technician cantransfer the five blastocysts indicated to be most likely to producefemale calves, in keeping with the goals of the dairy operation. In thisway, desirable characteristics (or characteristics desired to beavoided) in mammalian offspring produced from a videoed embryo can bepredicted for use in determining whether or not to transfer theparticular embryo (blastocyst) into a recipient.

Turning now to the microscopic nature of the morphokinetic movements ofembyros, not even under a microscope are such movements humanlydiscernible. For the purpose of revealing and sometimes accentuatingcertain aspects of embryo movement, Video Motion Magnification (VMM) hasbeen optionally employed in accordance with the teachings of thisdescription. VMM uses standard digital video image data sequences asinput and applies spatial decomposition followed by temporal filteringof the frames. The resulting signal is then amplified to reveal theembryo's morphokinetic movement that has heretofore been unobservable.VMM is a relatively new technology that has been utilized in selectphysiological diagnostic applications, including remote, non-invasivemeasurement of patient heart rates and infants' respiration. In essence,this software can take videos of seemingly static scenes and causemicro-motions captured therein to be rendered visually observable.

As has been well documented, developing embryos are undergoing mitosisat an exponential rate. Early in the development of this technology, itwas hypothesized that applying video motion magnification to real-timevideo image data of embryos might allow operators to directly observeembryo development, embryo decay, and detect the presence of geneticabnormalities, among other things. However, it came to be appreciatedthat application of VMM to embryo video image data can be an enhancementin certain situations to the AI/ML model creation and utilization asdisclosed herein. One of the most appealing aspects of VMM's processingof embryo video image data is its revelation of embryo movement that iseasily displayed for human observation. Still further, when embryo videoimage data is VMM pre-processed and then used to create AI/ML models asotherwise (without VMM treatment) described herein, the ensuing embryoanalysis and prediction can be enhanced.

In a study designed to determine the effectiveness of video motionmagnification to amplify previously undetectable embryo morphokinetics,videos of bovine blastocysts were recorded at 150× magnification for twominutes by a licensed veterinarian using an inverted microscope. Oncerecorded, the videos were “filtered” using the VMM software, Lambda Vue,and movement of both the inner cell mass and zona pellucida becamevisually discernible when displayed. Protrusions, bulges, depressions,pulses and changes in embryo shape became observable. The presentlydisclosed technology is the first known to optionally amplify, and thenanalyze recorded micro-motions and the morphokinetics of embryo growthand development utilizing real time video, and without the use of timelapse imaging. In this study, ImageJ software developed by the NationalInstitute of Health was utilized to take certain measurements of theembryos at specific locations and specific predetermined time intervalsto study changes in embryo shape and size over short periods of time asdescribed herein with respect to FIGS. 1-6.

Even if not used for predictive purposes, amplification of themicro-motions of an embryo can be instructive (and comforting) to anoperator as protrusions, bulges, depressions and shifts in the innercell mass, trophectoderm and zona pellucida are made visible.Heretofore, it has been known, but not perceivable on a real-time basis,that blastomeres are actively dividing and increasing embryo mass untildifferentiation occurs at the blastocyst stage of development. Then, theblastomeres begin to arrange themselves into a distinct inner cell mass(developing fetus) and trophectoderm (developing placenta). In thisregard, the human oocyte develops from a single cell into a blastocyst(˜300 cells) in a span of five to six days. Previously, changes inblastomere arrangement, size, cell count and orientation were onlydemonstrated using time-lapse photography.

Cells grow, but they also die, and in the process undergo certainphysical changes. When a cell dies, membrane proteins lose structuralintegrity and cannot osmoregulate their intracellular environment.Constituents from the external environment permeate into the cellthrough osmosis and intracellular constituents leak out of the cell.Non-viable embryos that include dead or dying cells absorb water asequilibrium is attempted. This occurs because water is attracted to thecell's intracellular environment and causes a low degree of cellular(embryo) swelling. After a cell has become non-viable for several hours,larger intracellular constituents such as salts and proteins begin toexit the cell and leak through the membrane proteins. This can beobserved as cellular fragmentation and degrading. These properties causephysical changes in the appearance of the embryo which occur over time.However, the minute physical changes (with respect to both distance,rate and duration of motion) associated with both healthy embryocellular growth and embryo cellular death are not humanly visible, evenwith the aid of a microscope up to 150× magnification. Applying videomotion magnification filtering to embryo real-time video image datarenders such motion visible to an operator and can optionally beincluded in the AI/ML modeling process described herein.

FIG. 7 depicts a series of twelve frames extracted from a video clip ofa blastocyst (having a frame rate of 2 fps), without magnification(arranged left to right, top to bottom). Though it is known thatblastocyst motion is occurring, it is not visible, even under theapplied 150× microscope magnification. However, FIG. 8 depicts the same12 frames after motion magnification, and the consecutive frames clearlydepict motion in that the frames are progressively different, indicativeand illustrative of the blastocyst's real-time motion. In at least oneinstance, video motion magnification has been used to detect embryo rateof growth to forecast cell count that is non-invasively predictive ofembryo sex.

Another benefit of motion amplification is that the amplified image datacan accentuate select motion characteristics being analyzed throughvector weighting and other techniques (via computer vision, digitalimage processing or otherwise) during the training and model developmentstage, or afterward during utilization of the generated model forpredictive purposes.

In support of the discoveries disclosed herein regarding training andusing AI/ML models to predict, among other things, embryo viability andcertain offspring characteristics, reference is made to the disclosureof FIG. 1. Therein, a proxy for each of several embryo's morphokineticactivity has been quantified and plotted on the illustrative graph. Forthe quantification, a mean of average subzonal change was determined foreach embryo, and which was considered to represent the embryo'smorphokinetic activity. Each such value was divided by the particularembryo's inner cell mass area to calculate a standardized percentmovement of that embryo, comparable with the other similarly“normalized” embryos. FIG. 1 represents a study in which thestandardized percent morphokinetic activity values for each of 94 bovineembryos were plotted on the displayed graph. From the graph it isgleaned that twenty-four embryos (24/94=25%) had morphokinetic changesoutside (some above, some below) of 2 standard deviations from the mean,which is signified by the left and right “tails” of the illustrativebell-shaped plot. Of those 24 embryos, 17 (17/24=70%) did not establishpregnancies (“open”); still further, these 17 embryos represent18%=17/94 of the 94 total embryos. This is confirmatory in that itcoincides with what has been found in practice and predicted by theinstant models; that is, approximately 20% of grade-1 and grade-2technician-assessed embryos, though otherwise expected, fail to producea pregnancy upon transfer.

FIG. 2 is a histogram that depicts, out of the same 94 bovine embryosplotted in FIG. 1, the standardized percent morphokinetic activityvalues for the 50 (of the 94) bovine embryos that produced pregnancy. Ofthese 50 embryos, 42 (84%) demonstrated morphokinetic activity valuesranging from 0.07%-0.23% (that is, taken from the left, the first threebars). However, the distribution of embryos across those first threebars—0.07%-0.12% {13 embryos}; 0.12%-0.18% {16 embryos} and 0.18%-0.23%{13 embryos}—were not statistically significant (p>0.05). Six embryos(12% of the 50 embryos that established pregnancies) demonstratedpercent morphokinetic activity values of 0.23%-0.29% and one embryo (2%of the 50 embryos that established pregnancies) demonstrated a percentmorphokinetic activity value of 0.29-0.34%. This is confirmatory of theposition that embryos with hyperactive morphokinetic activity are lesslikely to produce pregnancy than those embryos having moderatemorphokinetic activity, a position that has also been determined andpredicted by the models of the present disclosure.

The histogram of FIG. 3 illustrates and supports the proposition in thisdisclosure that determined inner cell mass range of motion within thesubzonal space is indicative/predictive of the embryo's competence forestablishing pregnancy. The histogram is based on measured radialdistances of the gap spaces between the outer contours of the inner cellmass area and the zona pellucida at twelve indicated equal-spacedlocations therearound every five seconds between 5 and 35 seconds (7measurements) along the dashed lines depicted in FIGS. 5 and 6. For eachof the 94 embryos, the measured amounts of motion over the 35 secondperiod at each of the 12 locations were averaged for each of the 94embryos that were transferred into recipients. The histogram of FIG. 3depicts the 50 embryos that achieved pregnancy. Therein, it is revealedthat embryos in the lower ranges of subzonal activity (bars toward theleft) established pregnancies at a higher frequency than embryos with ahigh range of subzonal activity (bars toward the right) (p<0.05). Out ofthe 50 embryos, 48 (96%) demonstrated average ranges of motion between1.6 um and 12.7 um, whereas only 2 out of the 50 embryos (4%)demonstrated a range of motion higher than 12.7 um.

FIG. 4 is a graph that depicts distances between the outer contours ofthe inner cell mass area and the zona pellucida of each of the 94embryos measured at the specific 12 locations indicated in FIGS. 5 and 6at five second intervals during a 35 second observatory period. Therange of motion was calculated at each of the 12 locations over the 35second duration and then averaged for each of the 94 embryos which weretransferred into recipients. Nine of the 94 embryos (9.57%) demonstratedan average range of subzonal motion (activity) greater than 12.5 um, andseven of those nine embryos did not establish pregnancies, againindicating that embryos experiencing a high degree of subzonal activityare less likely to establish pregnancies than embryos experiencing amoderate range of motion.

In another study, bovine oocytes were fertilized in vivo by a licensedveterinarian. The embryos were flushed from the donor and placed inholding media. Two-minute video recordings were taken on an invertedmicroscope at 150× magnification. Confirmatory measurements were takenabout each embryo using ImageJ Software. Among others, thosemeasurements include: (1) Zona pellucida X axis diameter; (2) zonapellucida Y axis diameter; (3) mass X axis diameter; (4) mass Y axisdiameter; (5) zona area; (6) mass area; (7) inner cell mass area and (8)rotational shift within the subzonal space. For each two-minute video,each of these measurements were made at the 0, 15, 30, 45, 60, 75, 90,105 and 120 second time points. Immediately after their video was taken,each embryo was transferred into a recipient animal as a singleton (oneembryo per recipient). Subsequent to this study, additional studiesobtaining similar results have been conducted on embryo video clips thatare 35 seconds long and were taken at ten frames per second (10 fps)prior to transfer into a recipient. Thereafter, pregnancy data, asoutcome data, was obtained by trans-rectal ultrasound at forty days posttransfer.

In still another study, 150 grade-1 and grade-2 embryos were flushedfrom beef cattle by a licensed veterinarian. Two-minute video recordingswere captured from each embryo with an Iolight® inverted microscope at150× in a standard culture of Vigro Holding Plus media. All embryos weretransferred into eligible recipients. Videos were processed with LambdaVue Video Motion Magnification Videoscope at 0.2-4 video frequency. Eachamplified video was then assessed using ImageJ measuring software bytaking measurements of the same embryo features at 15 second timeintervals. The following measurements were made on each embryo, every 15seconds: (1) Zona pellucida Y-axis diameter; (2) zona pelludica X-axisdiameter; (3) mass Y-axis diameter; (4) mass X-Axis diameter; (5) zonapelludica area and (6) inner cell mass area and at twelve locations onthe sub-zonal space as generally depicted in FIGS. 5 and 6. Observationswere also made of bulges on the zona pellucida and bulges on the innercell mass, when occurring.

In yet another trial, videos of embryos were obtained and amplifiedusing video motion magnification. Y-axis diameter of the inner cellularmass was measured using ImageJ software (National Institute of Health)and scaled to represent units in microns. Measurements were obtained atfive second intervals from 0 to 35 seconds. Quantifiable changes inembryo inner cell mass diameter were observed, demonstratingmorphokinetic changes in short observatory periods. These were observedvisually as bulges, depressions and protrusions in the inner cell massover time. These changes infer predictive information pertaining toembryo health and viability based on the other findings describedherein.

In a similar trial, videos of embryos were obtained and amplified usingvideo motion magnification. Area of the inner cell mass was obtainedusing measurement tools available from ImageJ software (NationalInstitute of Health) and scaled to represent units in microns.Measurements were obtained at five second intervals from 0 to 35seconds. As before, quantifiable changes in inner cell mass area wereobserved, likewise demonstrating morphokinetic changes in shortobservatory periods. The measurements represent a 2-D area of thevisible inner cell mass. It was observed that the cells were activelymoving, likely due to mitosis and cellular division. It is believed thatin some instances one cell mass overlapped another making the 2D area ofthe second cell mass appear to decrease, when it was not decreasing. Thepresent technology demonstrates active development or cellular decay anddoes not depend on additive or subtractive trends as in time-lapseanalysis. As revealed by the present technology, these cellular (embryo)changes evidence morphokinetic developmental characteristics that are infact predictive of embryo health, viability, stress, and geneticcharacteristics, both inferior and superior compared to average or“normal”.

This disclosure describes several different permutations of acomputer-implemented method for predicting an embryo's outcome byprocessing video image data of the embryo. The method includes receivingimage data derived from video of a target embryo taken at substantiallyreal-time frame speed during an embryo observation period of time. Thevideo contains recorded morphokinetic movement of the target embryooccurring during the embryo observation period of time and which isrepresented in the received image data. The received image data isprocessed using a model generated utilizing machine learning andcorrelated embryo outcome data.

In a further aspect, the disclosed method predicts a correlating embryooutcome for the target embryo. In this sense, the predicted “correlatingoutcome” is of the same nature as the “correlated outcome” used to trainthe model. For instance, if a model is trained on images (image data)known to be of cats (the “correlated outcome”), in most instances, thatso-trained model will be used to analyze (process) images (image data)and predict the likelihood (probability) the analyzed image is of a cat(the “correlating outcome”).

In one option, the present method predicts a likelihood the targetembryo will produce a specific sex offspring. In accordance with thisdisclosure, a trained model can be used to make such a prediction. In aninitial step, machine learning is used to train a model using image datafrom video of each of a number of embryos. For each embryo, it is knownwhether that embryo produced a male or female offspring and datarepresenting that fact is paired to the corresponding video. In thissense, the corresponding male/female data is “correlated outcome data”to the respective videos. That is, the male/female “outcome data” is“correlated” with the respective videos. Machine learning is thenutilized to analyze the paired data (video image data with its matchedoffspring sex data) and determine characteristics in the video imagedata that are indicative of male sex offspring and characteristics inthe video image data that are indicative of female sex offspring.Thereafter, to predict the likely sex of a produced offspring of aparticular target embryo, received image data from real-time video takenof that embryo is processed using the trained model. The model outputs a“correlating embryo outcome” which is a prediction of the likelihood thetarget embryo will produce a male sex offspring and/or a prediction ofthe likelihood the target embryo will produce a female sex offspring.

In general, the method described above for generatingmachine-learning-based models and utilizing such models to process dataand make predictions therewith, applies to the other predictivescenarios described herein.

As further options, the disclosed method can predict: (i) a likelihoodthe target embryo will produce human male offspring; (ii) a likelihoodthe target embryo will produce human female offspring; (iii) alikelihood the target embryo will produce bovine male offspring; or (iv)a likelihood the target embryo will produce bovine female offspring.

In another option, the method predicts a likelihood of successfultransfer of the target embryo into a recipient. In yet another option,the method predicts viability of the target embryo at the time oftransfer. In still another option, the method predicts a likelihood thetarget embryo will produce a genetically inferior offspring compared toaverage or “normal”. In another option, the method predicts a likelihoodthe target embryo will produce a genetically superior offspring comparedto average or “normal”.

According to this technology, a predicted embryo outcome of the targetembryo can be communicated to the originator of the received image datathereby assisting in a decision whether to transfer the target embryointo a recipient.

In one option, the correlated embryo outcome data represents embryotransfers into recipients that established pregnancy. In another option,the correlated embryo outcome data represents embryo transfers intorecipients that produced livebirth offspring. In still another option,the correlated embryo outcome data represents embryo transfers intorecipients that produced livebirth male offspring. In yet anotheroption, the correlated embryo outcome data represents embryo transfersinto recipients that produced livebirth female offspring. In stillanother option, the correlated embryo outcome data represents embryotransfers into recipients that produced genetically inferior offspringcompared to average or “normal”. In even another option, the correlatedembryo outcome data represents embryo transfers into recipients thatproduced genetically superior offspring compared to average or “normal”.

In a further aspect, the method includes receiving outcome dataindicative of whether successful transfer of the target embryo into arecipient occurred, where successful transfer is indicated by the targetembryo producing pregnancy in the recipient. In another aspect, themethod includes receiving outcome data indicative of whether successfultransfer of the target embryo into a recipient occurred, wheresuccessful transfer is indicated by the target embryo producinglivebirth offspring out of the recipient.

In one embodiment, the method includes utilizing a frame speedsufficiently fast to capture at least one individual embryomorphokinetic movement during the embryo observation period of time. Inanother embodiment, the method includes utilizing a speed sufficientlyfast to capture a series of embryo morphokinetic movements during theembryo observation period of time. In one option, the method utilizes aframe speed at least as fast as fifteen frames per second. In anotheroption, the method utilizes a frame speed of at least ten frames persecond. In yet another option, the method utilizes a frame speed of atleast two frames per second. In still another option, the methodmaintains substantially uniform frame speed during the embryoobservation period of time.

The presently described technology includes selecting a duration ofimage data received and determining the length of time of the embryoobservation period of time based there upon. In one option, the embryoobservation period of time is less than two minutes. In another option,the embryo observation period of time is less than thirty seconds. Inone aspect, the method includes selecting an embryo observation periodof time of less than two minutes. In another aspect, the method includesselecting an embryo observation period of time of less than thirtyseconds.

In one example, the presently described real-time video clips, fromwhich the received image data is obtained, are taken using an invertedmicroscope camera from a perspective below the target embryo. In anotherexample, the video from which the received image data is obtained istaken using an inverted microscope camera from a perspective below andthrough a petri dish containing the target embryo, and in which thebottom floor of the petri dish is substantially flat and clear enablingthe video of the target embryo to be optically accurate. In yet anotherexample, the video from which the received image data is obtained istaken using a smartphone camera. In still another example, the camera isdirectly connected to a circuit board of an inclusive platform.

The present disclosure describes a method of receiving, from the samevideo, image data of each of a plurality of target embryos. In oneembodiment, the method includes processing the image data for each ofthe plurality of target embryos utilizing the model and predicting anembryo outcome score for each of the plurality of target embryos. Inaccordance with this embodiment, the method includes communicating thepredicted embryo outcome score of each of a plurality of the targetembryos to an originator of the received image data, or otherwise userof the system, thereby assisting a decision of which embryos totransfer. Another embodiment includes communicating a ranking of thepredicted embryo outcome scores to an originator of the received imagedata thereby assisting a decision of which embryos to transfer. Inanother embodiment, the method includes transmitting display data to adisplay observable by an originator of the received image data thatcauses ranked embryo outcome scores to be displayed in a spatialarrangement on the display that corresponds to an existing spatialarrangement of the target embryos.

The present technology facilitates a method of amplifying at least aportion of image data from within the received image data thatrepresents morphokinetic movement of the target embryo. In oneembodiment, the morphokinetic movement of the target embryo is humanlyimperceptible at 150× magnification. In another embodiment, the methodutilizes Eulerian video magnification to amplify the portion of imagedata from within the received image data that represents morphokineticmovement of the target embryo. In a further embodiment, the methodutilizes the amplified data representing the morphokinetic movement ofthe target embryo to enable display of a moving image of the targetembryo in which the movement is humanly perceptible.

In one aspect, at least one characteristic of the target embryoevidenced in the received image data is elasticity of the embryo's outerwall.

The present disclosure includes the method of receiving secondary dataindicative of the target embryo's viability derived from a specificgravity assessment made of the embryo proximate in time to when thevideo from which the received image data is obtained was taken. In oneoption, the specific gravity assessment is made of the target embryowithin one hour of taking the video of the target embryo. In anotheroption, the specific gravity assessment is dependent upon a descent rateof the target embryo through embryo culture media.

In one embodiment, the method uses machine learning that comprisesutilization of an artificial neural network.

In another aspect, the present disclosure describes a system containingone or more processors and a computer-readable medium, instructionsstored within, which when executed by the one or more processors, causethe one or more processors to predict an embryo outcome by processingvideo image data of an embryo. The method includes receiving image dataderived from video of a target embryo taken at substantially real-timeframe speed during an embryo observation period of time. The video'sresolution is sufficient (high enough) to record and does recordmorphokinetic movement of the target embryo occurring during the embryoobservation period of time. The movement is represented in the receivedimage data and the received image data is processed using a modelgenerated utilizing machine learning and correlated embryo outcome data.

In still another aspect, the present disclosure describes anon-transitory computer-readable storage medium containingcomputer-readable instructions, which when executed by a computingsystem, cause the computing system to predict an embryo outcome byprocessing video image data of an embryo. The method includes receivingimage data derived from video of a target embryo taken at substantiallyreal-time frame speed during an embryo observation period of time. Thevideo's resolution is sufficient (high enough) to record and does recordmorphokinetic movement of the target embryo occurring during the embryoobservation period of time. The movement is represented in the receivedimage data. The received image data is processed using a model generatedutilizing machine learning and correlated embryo outcome data.

GENERAL DESCRIPTION OF AI/ML AND EXEMPLARY SYSTEM CONFIGURATION(S)

The disclosure set forth below provides general description of variousconfigurations of the subject technology and is not intended torepresent the only configurations in which the subject technology can bepracticed. The appended drawings are incorporated herein and constitutea part of this specification. This description includes details for thepurpose of providing a more thorough understanding of the subjecttechnology. However, the subject technology is not limited to thespecific details set forth therein and may be practiced with or withoutthese details. In some instances, structures and components are shown inblock diagram form so as to avoid obscuring the concepts of the subjecttechnology.

The disclosure now turns to additional discussion of models that can beused in the environments and techniques described herein. Specifically,FIG. 10 is an illustrative example of a deep learning neural network100. These networks are referred to as “neural” networks because theyreflect the behavior of the human brain. These neural networks, alsoreferred to as artificial neural networks (ANNs) and/or simulated neuralnetworks (SNNs), are subsets of machine learning (ML). The network hasan input layer 120 that is configured to receive input data, which inthe present case is the video image data of embryos. The neural network100 includes multiple hidden layers 122 a, 122 b, through 122 n. Thehidden layers 122 a, 122 b, through 122 n include “n” number of hiddenlayers, where “n” is an integer greater than or equal to one. The numberof hidden layers can be made to include as many layers as needed for thegiven application. The neural network 100 further includes an outputlayer 121 that provides an output resulting from the processingperformed by the hidden layers 122 a, 122 b, through 122 n. It is thepresence of the multiple hidden layers that gives rise to the “deeplearning” description.

The neural network 100 is a multi-layer neural network of interconnectednodes. Each node can represent a piece of information. Informationassociated with the nodes is shared among the different layers and eachlayer retains information as information is processed. In some cases,the neural network 100 can include a feed-forward network, in which casethere are no feedback connections where outputs of the network are fedback into itself. In some cases, the neural network 100 can include arecurrent neural network, which can have loops that allow information tobe carried across nodes while reading in input.

Information can be exchanged between nodes through node-to-nodeinterconnections between the various layers. Nodes of the input layer120 can activate a set of nodes in the first hidden layer 122 a. Forexample, as shown, each of the input nodes of the input layer 120 isconnected to each of the nodes of the first hidden layer 122 a. Thenodes of the first hidden layer 122 a can transform the information ofeach input node by applying activation functions to the input nodeinformation. The information derived from the transformation can then bepassed to and can activate the nodes of the next hidden layer 122 b,which can perform their own designated functions. Example functionsinclude convolutional, up-sampling, data transformation, and/or anyother suitable functions. The output of the hidden layer 122 b can thenactivate nodes of the next hidden layer, and so on. The output of thelast hidden layer 122 n can activate one or more nodes of the outputlayer 121, at which an output is provided. In some cases, while nodes inthe neural network 100 are shown as having multiple output lines, a nodecan have a single output and all lines shown as being output from a noderepresent the same output value.

In some cases, each node or interconnection between nodes can have aweight that is a set of parameters derived from the training of theneural network 100. Once the neural network 100 is trained, it can bereferred to as a trained neural network, which can be used to classifyone or more activities. For example, an interconnection between nodescan represent a piece of information learned about the interconnectednodes. The interconnection can have a tunable numeric weight that can betuned (e.g., based on a training dataset), allowing the neural network100 to be adaptive to inputs and able to learn as more and more data isprocessed.

The neural network 100 can be pre-trained to process the features fromthe data in the input layer 120 using the different hidden layers 122 a,122 b, through 122 n in order to provide the output through the outputlayer 121.

In some cases, the neural network 100 can adjust the weights of thenodes using a training process called backpropagation. As noted above, abackpropagation process can include a forward pass, a loss function, abackward pass, and a weight update. The forward pass, loss function,backward pass, and parameter update is performed for one trainingiteration. The process can be repeated for a certain number ofiterations for each set of training data until the neural network 100 istrained well enough so that the weights of the layers are accuratelytuned.

In general, and as noted above, for a first training iteration for theneural network 100, the output will likely include values that do notgive preference to any particular class due to the weights beingrandomly selected at initialization. For example, if the output is avector with probabilities that the object includes different classes,the probability value for each of the different classes may be equal orat least very similar (e.g., for ten possible classes, each class mayhave a probability value of 0.1). With the initial weights, the neuralnetwork 100 is unable to determine low level features and thus cannotmake an accurate determination of what the classification of the objectmight be. A loss function can be used to analyze error in the output.Any suitable loss function definition can be used, such as aCross-Entropy loss. Another example of a loss function includes the meansquared error (MSE), defined as E_total=Σ(½ (target-output){circumflexover ( )}2). The loss can be set to be equal to the value of E_total.

Generally, a goal of training is to minimize the amount of loss so thatthe predicted output is the same as the training label. The neuralnetwork 100 can perform a backward pass by determining which inputs(weights) most contributed to the loss of the network, and can adjustthe weights so that the loss decreases and is eventually minimized. Aderivative of the loss with respect to the weights (denoted as dL/dW,where W are the weights at a particular layer) can be computed todetermine the weights that contributed most to the loss of the network.After the derivative is computed, a weight update can be performed byupdating all the weights of the filters. For example, the weights can beupdated so that they change in the opposite direction of the gradient.The weight update can be denoted as w=w_i-η dL/dW, where w denotes aweight, w_i denotes the initial weight, and η denotes a learning rate.The learning rate can be set to any suitable value, with a high learningrate including larger weight updates and a lower value indicatingsmaller weight updates.

The neural network 100 can include any suitable deep network. Oneexample includes a convolutional neural network (CNN), which includes aninput layer and an output layer, with multiple hidden layers between theinput and out layers. The hidden layers of a CNN include a series ofconvolutional, nonlinear, pooling (for downsampling), and fullyconnected layers. The neural network 100 can include any other deepnetwork other than a CNN, such as an autoencoder, a deep belief nets(DBNs), a Recurrent Neural Networks (RNNs), among others.

As understood by those persons skilled in these arts, machine-learningbased classification techniques can vary depending on the desiredimplementation. For example, machine-learning classification schemes canutilize one or more of the following, alone or in combination: hiddenMarkov models; recurrent neural networks; convolutional neural networks(CNNs); deep learning; Bayesian symbolic methods; generative adversarialnetworks (GANs); support vector machines; image registration methods;applicable rule-based system. Where regression algorithms are used, theymay include but are not limited to: a Stochastic Gradient DescentRegressor, and/or a Passive Aggressive Regressor, and the like.

Machine learning classification models can also be based on clusteringalgorithms (e.g., a Mini-batch K-means clustering algorithm), arecommendation algorithm (e.g., a Miniwise Hashing algorithm, orEuclidean Locality-Sensitive Hashing (LSH) algorithm), and/or an anomalydetection algorithm, such as a Local outlier factor. Additionally,machine-learning models can employ a dimensionality reduction approach,such as, one or more of: a Mini-batch Dictionary Learning algorithm, anIncremental Principal Component Analysis (PCA) algorithm, a LatentDirichlet Allocation algorithm, and/or a Mini-batch K-means algorithmand the like.

The disclosure now turns to FIG. 11 which illustrates an example of aprocessor-based computing system 200 wherein the components of thesystem are in electrical communication with each other using a systembus 205. The computing system 200 can include a processing unit (CPU orprocessor) 210 and a system bus 205 that may couple various systemcomponents including the system memory 215, such as read only memory(ROM) 220 and random-access memory (RAM) 225, to the processor 210. Thecomputing system 200 can include a cache 212 of high-speed memoryconnected directly with, in close proximity to, or integrated as part ofthe processor 210. The computing system 200 can copy data from thememory 215, ROM 220, RAM 225, and/or storage device 230 to the cache 212for quick access by the processor 210. In this way, the cache 212 canprovide a performance boost that avoids processor delays while waitingfor data. These and other modules can control the processor 210 toperform various actions. Other system memory 215 may be available foruse as well. The memory 215 can include multiple different types ofmemory with different performance characteristics. The processor 210 caninclude any general-purpose processor and a hardware module or softwaremodule, such as module 1 232, module 2 234, and module 3 236 stored inthe storage device 230, configured to control the processor 210 as wellas a special-purpose processor where software instructions areincorporated into the actual processor design. The processor 210 mayessentially be a completely self-contained computing system, containingmultiple cores or processors, a system bus, memory controller, cache,etc. A multi-core processor may be symmetric or asymmetric.

To enable user interaction with the computing system 200, an inputdevice 245 can represent any number of input mechanisms, such as amicrophone for speech, a touch-protected screen for gesture or graphicalinput, keyboard, mouse, motion input, speech and so forth. An outputdevice 235 can also be one or more of a number of output mechanismsknown to those of skill in the art. In some instances, multimodalsystems can enable a user to provide multiple types of input tocommunicate with the computing system 200. The communications interface240 can govern and manage the user input and system output. There may beno restriction on operating on any particular hardware arrangement andtherefore the basic features here may easily be substituted for improvedhardware or firmware arrangements as they are developed.

The storage device 230 can be a non-volatile memory and can be a harddisk or other types of computer readable media which can store data thatare accessible by a computer, such as magnetic cassettes, flash memorycards, solid state memory devices, digital versatile disks, cartridges,random access memory, read only memory, and hybrids thereof.

As discussed above, the storage device 230 can include the softwaremodules 232, 234, 236 for controlling the processor 210. Other hardwareor software modules are contemplated. The storage device 230 can beconnected to the system bus 205. In some embodiments, a hardware modulethat performs a particular function can include a software componentstored in a computer-readable medium in connection with the necessaryhardware components, such as the processor 210, system bus 205, outputdevice 235, and so forth, to carry out the function. For clarity ofexplanation, in some instances the present technology may be presentedas including individual functional blocks including functional blockscomprising devices, device components, steps or routines in a methodembodied in software, or combinations of hardware and software.

In some embodiments the computer-readable storage devices, mediums, andmemories can include a cable or wireless signal containing a bit streamand the like. However, when mentioned, non-transitory computer-readablestorage media expressly exclude media such as energy, carrier signals,electromagnetic waves, and signals per se.

Methods according to the above-described examples can be implementedusing computer-executable instructions that are stored or otherwiseavailable from computer readable media. Such instructions can comprise,for example, instructions and data which cause or otherwise configure ageneral-purpose computer, special purpose computer, or special purposeprocessing device to perform a certain function or group of functions.Portions of computer resources used can be accessible over a network.The computer executable instructions may be, for example, binaries,intermediate format instructions such as assembly language, firmware, orsource code. Examples of computer-readable media that may be used tostore instructions, information used, and/or information created duringmethods according to described examples include magnetic or opticaldisks, flash memory, USB devices provided with non-volatile memory,networked storage devices, and so on.

Devices implementing methods according to these disclosures can comprisehardware, firmware and/or software, and can take any of a variety ofform factors. Typical examples of such form factors include laptops,smart phones, small form factor personal computers, personal digitalassistants, rackmount devices, standalone devices, and so on.Functionality described herein also can be embodied in peripherals oradd-in cards. Such functionality can also be implemented on a circuitboard among different chips or different processes executing in a singledevice, by way of further example.

The instructions, media for conveying such instructions, computingresources for executing them, and other structures for supporting suchcomputing resources are means for providing the functions described inthese disclosures.

Although a variety of examples and other information was used to explainaspects within the scope of the appended claims, no limitation of theclaims should be implied based on particular features or arrangements insuch examples, as one of ordinary skill would be able to use theseexamples to derive a wide variety of implementations. Further andalthough some subject matter may have been described in languagespecific to examples of structural features and/or method steps, it isto be understood that the subject matter defined in the appended claimsis not necessarily limited to these described features or acts. Forexample, such functionality can be distributed differently or performedin components other than those identified herein. Rather, the describedfeatures and steps are disclosed as examples of components of systemsand methods within the scope of the appended claims.

1. A computer-implemented method for predicting an embryo outcome byprocessing video image data of an embryo, the method comprising:receiving image data derived from video of a target embryo taken atreal-time frame speed during an embryo observation period of time,wherein said video contains recorded morphokinetic movement of thetarget embryo occurring during the embryo observation period of time andwherein said recorded morphokinetic movement is represented in thereceived image data; processing the received image data using a modelgenerated utilizing machine learning and correlated embryo outcome data;and predicting a likelihood the target embryo will produce a humanfemale offspring.
 2. (canceled)
 3. The method of claim 1, furthercomprising: predicting a likelihood of successful transfer of the targetembryo into a recipient.
 4. The method of claim 1, further comprising:predicting viability of the target embryo at the time of transfer. 5.The method of claim 1, further comprising: predicting a likelihood thetarget embryo will produce a genetically inferior offspring.
 6. Themethod of claim 1, further comprising: predicting a likelihood thetarget embryo will produce a genetically superior offspring.
 7. Themethod of claim 1, further comprising: communicating a predicted embryooutcome of the target embryo to an originator of the received image datathereby assisting a decision whether to transfer the target embryo. 8.The method of claim 1, wherein the correlated embryo outcome datarepresents embryo transfers into recipients that established pregnancy.9. The method of claim 1, wherein the correlated embryo outcome datarepresents embryo transfers into recipients that produced livebirthoffspring.
 10. The method of claim 1, wherein the correlated embryooutcome data represents transfers into recipients that producedlivebirth female offspring.
 11. The method of claim 1, wherein thecorrelated embryo outcome data represents embryo transfers intorecipients that produced genetically inferior offspring.
 12. The methodof claim 1, wherein the correlated embryo outcome data represents embryotransfers into recipients that produced genetically superior offspring.13. The method of claim 1, further comprising: receiving outcome dataindicative of whether successful transfer of the target embryo into arecipient occurred, wherein successful transfer is indicated by thetarget embryo producing pregnancy in the recipient.
 14. The method ofclaim 1, further comprising: receiving outcome data indicative ofwhether successful transfer of the target embryo into a recipientoccurred, wherein successful transfer is indicated by the target embryoproducing livebirth offspring out of the recipient.
 15. The method ofclaim 1, further comprising: utilizing a frame speed sufficiently fastto capture at least one individual embryo morphokinetic movement duringthe embryo observation period of time.
 16. The method of claim 1,further comprising: utilizing a frame speed sufficiently fast to capturea series of embryo morphokinetic movements during the embryo observationperiod of time.
 17. The method of claim 1, further comprising: utilizinga frame speed at least as fast as fifteen frames per second.
 18. Themethod of claim 1, further comprising: utilizing a frame speed at leastas fast as ten frames per second.
 19. The method of claim 1, furthercomprising: utilizing a frame speed at least as fast as two frames persecond.
 20. The method of claim 19, further comprising: maintainingsubstantially uniform frame speed during the embryo observation periodof time.
 21. The method of claim 1, further comprising: selecting aduration of image data received and thereby determining a length of timeof the embryo observation period of time.
 22. The method of claim 1,wherein the embryo observation period of time is less than two minutes.23. The method of claim 1, wherein the embryo observation period of timeis less than thirty seconds.
 24. The method of claim 1, furthercomprising: selecting an embryo observation period of time less than twominutes.
 25. The method of claim 1, further comprising: selecting anembryo observation period of time less than thirty seconds.
 26. Themethod of claim 1, wherein the video, from which the received image datais obtained, is taken using an inverted microscope camera from aperspective below the target embryo.
 27. The method of claim 1, whereinthe video, from which the received image data is obtained, is takenusing an inverted microscope camera from a perspective below and througha petri dish containing the target embryo and wherein a bottom floor ofthe petri dish through which the video is taken is substantially flatand clear thereby enabling the video of the target embryo to beoptically accurate.
 28. The method of claim 1, wherein the video fromwhich the received image data is obtained is taken using a smartphonecamera.
 29. The method of claim 1, further comprising: receiving, fromthe same video, image data of each of a plurality of target embryos. 30.The method of claim 29, further comprising: processing the image datafor each of the plurality of target embryos utilizing the model andpredicting an embryo outcome score for each of the plurality of targetembryos.
 31. The method of claim 30, further comprising: communicatingthe predicted embryo outcome score of each of a plurality of the targetembryos to an originator of the received image data thereby assisting adecision of which embryos to transfer.
 32. The method of claim 30,further comprising: communicating a ranking of the predicted embryooutcome scores to an originator of the received image data therebyassisting a decision of which embryos to transfer.
 33. The method ofclaim 32, further comprising: transmitting display data to a displayobservable by an originator of the received image data that causesranked embryo outcome scores to be displayed in a spatial arrangement onthe display that corresponds to an existing spatial arrangement of thetarget embryos.
 34. The method of claim 1, wherein the morphokineticmovement of the target embryo is humanly imperceptible at 150×magnification.
 35. The method of claim 1, further comprising: amplifyingat least a portion of image data from within the received image datathat represents morphokinetic movement of the target embryo.
 36. Themethod of claim 35, further comprising: utilizing Eulerian videomagnification to amplify the portion of image data from within thereceived image data that represents morphokinetic movement of the targetembryo.
 37. The method of claim 35, further comprising: utilizing theamplified data representing the morphokinetic movement of the targetembryo to enable display of a moving image of the target embryo in whichthe movement is humanly perceptible.
 38. The method of claim 1, whereinat least one characteristic of the target embryo evidenced in thereceived image data is elasticity of the embryo's outer wall.
 39. Themethod of claim 1, further comprising: receiving secondary dataindicative of the target embryo's viability and wherein the secondarydata derives from a specific gravity assessment made of the embryoproximate in time to when the video from which the received image datais obtained was taken.
 40. The method of claim 39, wherein the specificgravity assessment is made of the target embryo within one hour oftaking the video of the target embryo.
 41. The method of claim 39,wherein the specific gravity assessment is dependent upon a descent rateof the target embryo through embryo culture media.
 42. The method ofclaim 1, further comprising: utilizing machine learning that comprisesutilization of an artificial neural network.
 43. A system comprising:one or more processors; and a computer-readable medium comprisinginstructions stored therein, which when executed by the one or moreprocessors, cause the one or more processors to: predict an embryooutcome by processing video image data of an embryo, said predictioncomprising: receiving image data derived from video of a target embryotaken at real-time frame speed during an embryo observation period oftime, wherein said video contains recorded morphokinetic movement of thetarget embryo occurring during the embryo observation period of time andwherein said recorded morphokinetic movement is represented in thereceived image data; processing the received image data using a modelgenerated utilizing machine learning and correlated embryo outcome data;and predicting a likelihood the target embryo will produce a humanfemale offspring.
 44. A non-transitory computer-readable storage mediumcomprising computer-readable instructions, which when executed by acomputing system, cause the computing system to: predict an embryooutcome by processing video image data of an embryo, said predictioncomprising: receiving image data derived from video of a target embryotaken at real-time frame speed during an embryo observation period oftime, wherein said video contains recorded morphokinetic movement of thetarget embryo occurring during the embryo observation period of time andwherein said recorded morphokinetic movement is represented in thereceived image data; processing the received image data using a modelgenerated utilizing machine learning and correlated embryo outcome data;and predicting a likelihood the target embryo will produce a humanfemale offspring.